somewhere to talk about random ideas and projects like everyone else

stuff

#google wave

HTTP based federated protocol for real time hierarchical message manipulation 29 August 2010

In other words. It’s like google wave, but simpler in every possible way.

This protocol uses two servers. The federation server and the storage server. The latter is incredably simple. In fact, the reference implementation is only about 200 lines of JS (Node.JS FTW). Thats because a storage server accomplishes just about three things. It receives message deltas. It applies them. And it pushes the delta to all subscribers. The subscribers are the federation servers, they act on the behalf of multiple clients, keeping track of users, their inboxes, etc.

Anyway, the big part about the design is that there is only one unit of information, and that is the message. There’s no such thing as waves, wavelets, conversations, private replies, threads (sort of lying here), blips and other information. It’s just messages. Messages are stored on storage servers, and are filled with HTML and a tree of information.

Messages can have other messages inside them. It’s just some xml-ish stuff. <thread></thread> is a thread and <message name=”http://blahblahblah.com/blahblahblah"></message&gt; is a message that goes inside the thread. You can stick it anywhere. In the middle (inline replies!) or at the end (normal replies!).

Messages don’t even have to have text. Gadgets are just messages that are slightly different.

http://github.com/antimatter15/awesomeness

There’s a lot that it doesn’t do because I’m too lazy to do it. And I can’t give you a live demo because I don’t have a node-enabled server.



Wave Reader 30 June 2010

Information 2 - Chromium_080
Wave Reader started around early December of 2009 to fill a somewhat significant void. Embedded waves could not be accessed by people who were not logged into their Google accounts, there was no way to link to a specific blip and the official wave client was insanely slow. Most of these issues have been mitigated, so the project is virtually defunct.

μwave updates again 30 June 2010

microwave-screen630-wave

Edit mode is no longer experimental, a new implementation includes a tiny diff engine which allows editing a post without necessarily destroying layout. Root blip editing is now possible. There is a new tag list on the bottom of each wave, also including an “Add Tag” button. Search results are now formatted with modification date, number of blips, number of unread blips and read/unread state. There is a new settings panel when you click the logo. Added support for the internet exploder browser starting at version six. Owner_utils is a setting which adds utilities like “set everyone as read only”. The New Wave feature no longer creates pop-up prompts, but rather silently creates and opens an empty wave. It renders the live-editing cursors. There is a new multipane interface for desktop. Gadget support has greatly improved. It handles rotation on a mobile webkit device better. It now uses Wave Data/Robots Protocol 0.22 and renders using the newly exposed conversation model.


Deep Integration Wave Embed API 30 June 2010

Just to prove that it

Can you tell that the screenshot on the left is actually an embedded wave? Probably not.

First, a little overview on what this is. A lot of people are interested in using Google Wave as a commenting system, however the official embed api at the current state suffers from several shortfalls. It’s tedious to operate a wave-integrated commenting system. As an author, you have to manually create a wave for every post, copy the content over to the wave, get the wave ID, and paste it into your template. Every time you change your post, you have to yet again manually sync the page with wave. Once you’ve gotten the wave into the page, the embedded wave itself has a few issues. You can’t style it using CSS, and you can only change the background color and font. The width can’t adjust to the width of your page after the initial load. There’s no way to get rid of or change the style of the huge blue border around it. It forces it’s own scroll bar which looks nothing like anything else on the page and prevents intuitive navigation of comments using the same scrollbar as the rest of the page. Lastly, there’s no way for anonymous readers to comment on the post.

For each of these issues, the new w2_embed api offers a solution. Instead of using iframes to hack another page onto the current one, this one is built into the page, injecting elements directly onto the page’s DOM. These elements can be styled in nearly any imaginable way through CSS and HTML. The sandboxed and namespaced single-function API allows for multiple waves on one page with less than 10KB of JS overhead (Compare to about 1.4 megabytes). Instead of relying on a special Wave ID, the app uses a centralized and remotely stored database which associates any identifier you provide with functional wave IDs. That means, that instead of using a string like googlewave.com!w+Blsm0RMW_A, you have the URL of your blog, say 2010/06/wave-embed-api. Not only that, but it can automatically pull the title of your wave, the authors, the permalink and tags from the post, automatically create a wave from that with the same tags and title with the content automatically set as the root blip as well as a permalink back to the blog post. When viewed from wave, the user can follow, unfollow, search and find waves using the exact same familiar interface. On your blog, you can make the wave backend entirely seamless and subtle. Since it’s built on top of Anonybot, people can reply blips without having google wave accounts right from your blog.

It’s all part of a new embed API built upon Anony-bot (which itself is built upon microwave, which is built on wave reader, which is built upon the prototype python desktop client… Someone should make a wordpress plugin for this to make this family history even longer).

The last post noted the beginning of this project, and here is part two. The API has been cleaned up a lot. The code has been simplified. Features added and design changed.

The major change is a total decoupling of it and the mainline Anonybot codebase. There are some major implications of the change. In the original prototype of the API, it still had the iconic blip context menu inherited from microwave. That’s gone. So is the participants bar, link to real wave client, blip html, blip content, reply box, and everything. There are now no required or automatically added html/dom elements. Everything is specified through a simple HTML templating engine. Since the code isn’t shared, changes from each individual project now on probably won’t be instantly applied to both. This has advantages and disadvantages. On the plus side, is that changes to anonybot won’t break the embed api. However, bug fixes on one may not propagate to the other. The code can be much smaller since features like search, which was previously hidden, can now be totally removed. Also, since it’s been rewritten from scratch as an embed api, it no longer exposes any extra globals and multiple embeds can be done for each page.

The biggest change is the templating engine. Before, it only worked with the blip design, but now it’s totally integral to every aspect of the interface. Lets walk through the basics of the API, starting with a minimal example.

All the content is wrapped in a huge hidden `div`, and inside is one element with the `wavebody` class. That huge wave is the scope of the templating engine. That means that if you put `{{1+1}}` anywhere inside the `div`, it’ll show up as 2. Place it anywhere outside the div, and it shows up as `{{1+1}}`. Simple, right? The `wavebody` element is where the actual blips will go. But since there aren’t any blips beforehand, `wavebody` is filled with the template for a single blip. The content gets multiplied several times and that makes the wave. The template scheme is really simple. Basically, you just wrap arbitrary JS code within `{{` and `}}`. The code gets executed (in the scope of the wave) and the block gets replaced with the return value of the script. When you’re inside `wavebody` there are a few additional blip-specific things that you can add. Now we have to learn the magical snippets of code you can use inside the curly brackets. One of the objects exposed to the templates is `wave`. The nice thing is that it mirrors [the wave api](http://code.google.com/apis/wave/extensions/robots/protocol.html#Wavelets). So the wave object exposes the following attributes (copied from the google page). * `creationTime` denotes the [Unix time](http://en.wikipedia.org/wiki/Unix_time) at which this wavelet was created. * `creator` denotes the address of the participant who created this wavelet. * `lastModifiedTime` denotes the [Unix time](http://en.wikipedia.org/wiki/Unix_time) at which the wavelet was last modified by any participant. * `participants` contains an array of participant IDs for all participants on the wave. * `rootBlipId` contains the Blip ID of the root blip. * `title` contains the title of the wavelet, which by default consists of the first line of text up to the first carriage return. * `version` contains the version of this wavelet. Each atomic operation on a wavelet increases this version number. * `waveId` contains the Wave ID of this wavelet. * `waveletId` contains the wavelet ID of this wavelet. Note that for waves which contain only one wavelet (that don’t have private conversations, in other words), this wavelet ID is usually of the form `conv+root` indicating that the wavelet is identical to the conversation root, the root wave. * `dataDocuments` contains a dictionary (associated array) of the IDs and data of any data documents attached to this wavelet. In addition, there are a few non-standard components. * `permalink` contains a link to the waveref which points to the specific wave on the real wave client * `blips` contains a js object (similar to a hash or dictionary) of blipIDs associated to blip objects * `bliparr` contains an array of blip ids in no particular order, most frequently used for getting the number of replies in a conversation in `{{wave.bliparr.length-1}}` There are also some functions you can call. For example, `wave.creationTime` is in the form of a unix timestamp. That probably isn’t too useful to an ordinary user, so a `format_time` function is there to format it like “6/30 10:45am”. If you want to have some other custom datetime format, get me a script to do it in javascript in less than 500 bytes and I’ll include it in the next version. * `format_time(int)` Format a unix-style date as a human-readable “6/30 10:45am” or if you prefer “m/d h:MMtt” * `render(blip)` Render a blip and return the HTML. Most of the time, it’s unnecessary as there’s a blip.html attribute There are a few actions that you can do. Certain strings like `{{addparticipant}}` can be used in `onclick=` attributes to trigger certain actions on various events.

Here you see a link and a button. The link has been configured to prompt the user for a participant ID to add when clicked, and the button was set to prompt the user for a tag to add when clicked. There are several such actions.

  • addparticipant Prompt the suer for a wave address to add to the wave
  • reload Reload the current wave
  • addtag Prompt the user for a tag to add
  • setname Prompt the user to set a username for posting There is also the user list. It comes in two flavors, {{participantlist}} and {{contributorlist}}. The latter only works when inside a blip scope (the stuff inside wavebody). They’re special because they’re actually interactive. When the list is long, it shortens it and creates a link to expand.

  • participantlist A list made from wave.participants

  • contributorlist Only works when in wavebody, a simple users list made from blip.contributors Now, is finally the stuff that goes in the blip. Like wave it extends some wave api features, so I’ll start with pasting what google has documented.

  • blipId contains the ID of blip in which the event occurred.

  • childBlipIds contains an array of blip IDs for each of the blip’s children.
  • contributors denotes participants who have contributed to the state of this blip.
  • creator denotes the participant who created this blip.
  • lastModifiedTime denotes the Unix time at which this blip was last modified by any participant.
  • content contains the textual content of this blip.
  • version contains the version of this blip. Each atomic operation on a blip increases this version number.
  • waveId contains the Wave ID associated with this blip.
  • waveletId contains the wavelet ID associated with this blip. Note that for waves which contain only one wavelet (that don’t have private conversations, in other words), this wavelet ID is usually of the form conv+root indicating that the wavelet is identical to the conversation root, the root wave. There are a few additional methods which, again are non-standard.

  • permalink contains a link to the waveref which points to the specific blip on the real wave client

  • html contains a string of the rendered (with formatting and annotations applied) content There are also actions, similar to the global addparticipant, reload and setname actions. They can still be used within wavebody but there are some actions which can only work in the blip template.

  • edit Add an edit box immediately below the blip, pre-populated with the text content of the blip

  • remove Remove the blip, it will first issue a confirmation prompt to the user
  • reply Add a reply box immediately below the blip. If there is already a blip below, the reply box is indented. Below is a little example showing how to use the blip object. Lastly is the JS part. Without it, nothing would ever show up. You need to include the currently 6.5KB JS library from http://anony-bot.appspot.com/assets/embed2.mini.js. After that, you need a separate script tag anywhere after the big hidden super element. It must be outside the super element (unimaginable evils will happen if it isn’t). Inside that script tag needs to be a function call to w2_embed. The first (and only) argument has to be a JS object with at least the element key pointing to a DOM element which contains an element with the class wavebody. There are lots of other config options, and here they are.

  • element (Required) a dom element which contains a div with the “wavebody” class

  • identifier The identifier for the wave that the server will use to generate a wave ID from, default: location.pathname
  • root_title If the identifier has not previously been associated with a wave ID, the wave will be created with the specified title. default: (no title)
  • root_content If the identifier has not previously been associated with a wave ID, the wave will be created with the specified content as the root blip. default: (no content)
  • participants The participants the wave should be created with if not previously associated with a wave ID, default: [], suggested: [‘public@a.gwave.com’]
  • hideroot Boolean whether or not to not render the root blip
  • edit A reference to a DOM element which can be templated into the edit box, should contain handlers for the actions {{submit}} and {{cancel}} as well as a textarea with the class “wavetext”
  • tags Set of tags to be added to the wave if the identifier has not previously been associated with a wave and must be created.
  • api_root The domain containing the server component (proxy and dictionary server). Default: anony-bot.appspot.com
  • gadgets Enable native gadgets (does not work)
  • render_state Display the user a gadget state
  • chronological Render blips chronologically (see Microwave’s “Classic Forum Layout” option)
  • json_url Location of JSON implementation to be loaded dynamically in case it’s not native to the browser default: http://anony-bot.appspot.com/assets/json2.mini.js
  • waveid Skip the identifier lookup and directly reference a wave by it’s id  

You can see that the function call was saved to a variable named demo. That’s useful because you can use that to reference later to manipulate things. The return of a w2_embed call is a JS object with the following keys.

  • config A reference to the initial config object
  • reply(text) Create a root-level reply
  • reload() Reload the wave. So lets try putting everything together.

 

Afterthoughts: Just a little thing to add at the end. You can can create a a div with the class waveedit as used above. It’s not strictly necessary, but it lets you customize things a bit more. You also need a CSS style for .wavethread with a padding-left (or margin-left) in order for threads to show up properly. Blips are automatically given a .waveblip class.

That screenshot on the top of the post, if you want to try it out. It’s here.

I’m not making any money off of this, so there’s no reason for me to hide the fallibilities of this solution from anyone. This, like any other solution is not perfect. There is no true wave-style real-time editing, it uses polling. It does however update every 10 seconds if it’s configured to do so using the poll_updates: true configuration option. It can’t ever really do much better, since app engine doesn’t yet support pushing out real time data to attached clients, even though they announced the neat Channel API (still waiting to use it!) at I/O. Either way, that probably won’t help too much as w2_embed uses JSONP for client/server communication across multiple domains, and it’s unlikely that the channel api will allow for that. Also, since there’s no way to fetch a single blip, the user must request the entire wave to check the tiny 32 bit version integer if it’s been incremented, and if it has, then the whole wave has to be fetched again. It’s by no means efficient, and so my server would be very very sad if the polling interval was significantly increased. It’s not very SEO friendly. Though it’s not always that people want the comments to be indexed by search engines, so some people may care less. Accessibility-wise, it shouldn’t be too horrible if the screen reader/browser supports javascript and hugely depends on the HTML being used in conjunction with the script.

As of date, this was my longest blog post ever, and I would be sad if nobody commented (ironically using my non-wave-based commenting system). And I’m not posting my source code unless someone comments, so, an incentive :P


μwave updates 03 June 2010

Microwave June 2 Search

Over a few days, things can change fairly quickly. There have been several speed improvements, a new Forum-Style blip rendering option which arranges blip linearly by the time edited with each containing a formatted quote of the parent to establish context. Attachments are now fully supported, including thumbnails and download links. The operations engine was totally rewritten which uses asynchronous XMLHttpRequest, a new callback based system and support for a batch operations (which means fewer requests and faster responses). A wavelet header containing a list of all participants in the entire wave has been added, as well as an Add Participant button. A specialized, extremely fast gadget viewer was added, which allows for blazingly fast rendering of two popular gadgets (and more will come), it works by bypassing the entire gadget infrastructure and loading trusted code directly inline with the DOM. There is a “New Wave” button which allows people to create new waves directly from the client. The OAuth backend was authenticated with google, for more secure login transactions. Blips have a new context menu which allows for features such as Delete Blip, Edit Blip and Change Title. A full changelog can be found here.

Try it out.


μwave lightweight mobile wave client 29 May 2010

Microwave Search-View May 29

I like bragging when I do something nobody else has done before. And μwave is the first true third-party wave client which is compatible with Google Wave. It’s free to use at http://micro-wave.appspot.com/ and works great on mobile devices. It supports searching for waves, opening them and writing replies.

Currently it doesn’t know read/unread state of the waves from the search panel and doesn’t know read/unread blips, but as of time of writing, its a limitation and flaw in the current version of the Google Wave Data API (introduced just ten days ago at Google I/O). Expect this to be resolved in the near future with upcoming versions of the API and this application.

The source code for the server component is open-source and can be found on github (though it’s slightly outdated, but the important stuff is there). It’s fairly simple (It’s based on the original example code so I’m going to have the same MIT license), but one of the few python scripts which can do authentication with google and pass commands to the data api. It relies on the Python OAuth Library.

The Blip renderer component of this is licensed under the MIT license and can be found on the old microwave repo. The only part left is the interface, which is going to be the usual GPLv3.

For a little bit of history, this isn’t exactly a new project. The google code project has existed since January 9th, 2010. The purpose was to create a mobile-friendly version of Wave Reader. But that goes even deeper, I can trace it back to the original Static-Bot dated to October 18, 2009. Then, the Google Wave embed API allowed people to view waves only if someone had a Google Wave account and was logged in at the time. This was quite problematic as Wave was still a limited preview which not many people had and probably hampered adoption.

Another separate but eventually convergent issue which led to the microwave project was “Desktop Wave Reader + GWave Client/Server Protocol“ post which I made on October 29th of 2009.

During late October of last year, I reverse-engineered some of the features of the Google Wave client. Up until then, the only published specs were the federation protocols, which dealt with how multiple wave servers would use a common protocol to allow multiple users without a central authority and for the gadget and robot apis. Notably missing was a client/server api, for a user of specifically the google wave client, which did not yet support federation (and to date, preview still does not), and to browse/view the waves in one’s inbox without needing to switch to an entirely new provider. The first component was the ability to read waves. After that was accomplished, I tried to reverse engineer a more complex aspect of the protocol, which was the ability to search waves. I eventually realized that that component, search was part of a larger puzzle, which was the real-time BrowserChannel wire protocol which virtually all of wave was based. I made some progress, but near the end, I gave up in frustration. Luckily, someone else became interested in the same thing, and Silicon Dragon basically got search working.

This happened now in early December. I started on a project called Wave Reader, which merged the ideas of static-bot with the desktop wave reader and a new functional blip rendering engine. At that time, the Google Wave client was still horrendously slow, taking several minutes at times to load large waves.

On January, I began a project to merge Wave Reader and the wire protocol (search). I thought an awesome name would be microwave (or μwave) and started the code repo on January 9th. I worked on it a bit, so that it was mostly complete, with search and loading all working, with one missing component: login. Eventually, I got bored and the project lay abandoned for a few months.

This gets us to basically 4 days ago, when I started working on a renaissance of the μwave project, based on the recently released Google Wave Data API. The first component was creating a new blip renderer specifically designed for parsing the new (much cleaner) json format which is part of the robots api. Then I created a client around that and created a python backend for having it work on app engine.

The Future is always awesome to prophesize about. In the coming weeks or days, google will probably update the data api to allow for information like So, while http://micro-wave.appspot.com will likely remain free and maintained for the forseeable future, I do plan on making a paid iPhone/iPad app. The iPhone app may have some extra features like offline/caching support.


Wave Reader 4.6 - Insanely Fast Edition 14 January 2010

Loading a 500 blip wave in Google’s GWT Client takes 3:34 to get to a usable state (Where the scroll bar works) on a 3ghz Core2 Duo (whose extra core admittedly won’t do much). It also uses 972 MB of RAM.

Loading the same wave in my Wave Reader, takes 678 milliseconds. A 315x speed-up. Also, my client is totally unoptimized, pure 30KB of javascript. On top of those features, anyone can view waves, without a google account, individual blips can be linked to, it supports rendering almost everything Wave can, that is gadgets, inline replies, nesting, font color/size, italics, bold, everything you could probably expect. Interestingly, when you add an attachment to Wave, and delete the parent blip, it still stores the attachment on the current wave state, and this client can display/link to them without using Playback. There is an option to generate plain simple HTML for turning a Wave into a standalone page or Website. Private waves can be exposed read-only as a website simply by adding the gwavereader@googlewave.com username.

Using it is simple, take a Wave https://wave.google.com/wave/#restored:wave:**googlewave.com!w%252Br5lewFqCA** and then put it after the Wave Reader URL http://antimatter15.com/misc/wave/read.html?**googlewave.com!w%252Br5lewFqCA** And magically you have a super awesome URL to link to.

You can learn some tricks on how to use it to do some more awesome things http://antimatter15.com/misc/wave/read.html?googlewave.com!w%252BrnG0vaFXA such as the before mentioned HTML generation.

Samples (Some random waves): http://antimatter15.com/misc/wave/read.html?googlewave.com!w%252Br5lewFqCA (New for 4.6 Inline reply support) http://antimatter15.com/misc/wave/read.html?Ze3l0mj0A http://antimatter15.com/misc/wave/read.html?oPg9HfEXE&beta http://antimatter15.com/misc/wave/read.html?Mu9eK7j2H http://antimatter15.com/misc/wave/read.html?AhbL5fooD&beta http://antimatter15.com/misc/wave/read.html?googlewave.com!w+UDMZOGpSG


Wave Unread Navigator 13 December 2009

I made an extension which shows a green sign (like gmail) on the side whenever there are blips outside the viewport. It allows you to quickly and effectively use keyboard navigation without blindly checking above and below, or generally scroll without needing to guess the position.

It comes as a Chrome Extension and also as a Greasemonkey Userscript.


Wave Gadget API issues (again) 07 September 2009

So I’ve been having issues with the Wave Gadget API (again). This one is simple, Wave isn’t really real time. So right after doing a submitDelta, you can’t get() the data and expect to have the new one instantly.

It’s been giving me some problems, but I’m getting around it by using my awesome wave gadget library which will now magically apply the changes immediately so you can access it even before things actually happen.

Also, partially due to this, things would be far more useful if you could get the date of each insertion or deletion.


Wave2 A higher level Wave Gadget State API 17 August 2009

I was working on a new little Wave State API update. It supports lists in the form of subkeys, and something very much like hierarchy and events on specific sub-nodes. This way you don’t have everything update (someone’s edits on frame 36 doesn’t matter if you’re on frame 42).

Everything is namespaced under a singleton global, wave2. It includes functions like

  • listen (execute a callback when something beginning with the prefix is changed)
  • ignore (un-listen),
  • keys (shortcut for wave.getState().getKeys()),
  • subkeys (get keys which begin with a certain prefix, important to the pseudo hierarchy),
  • set (submitdelta with first arg as name and second as value),
  • get (shortcut for wave.getState().get()
  • reset_gadget (a simple way to empty all the data in the store) And since it’s quite short, I’m putting it under public domain at http://antimatter15.com/misc/wave/wave2.js

Working on interface for Animator/Wave 14 August 2009

Interface of Ajax Animator as Google Wave Gadget

The name field will be replaced with buttons, and zoom may or may not be scrapped. Since gadgets only take up a fraction of the page (usually) it’s designed with size in mind, which may help on other space-constrained devices like netbooks.

It may have a cool feature like VectorEditor’s demo where you can draw off the canvas area.

As for why the tabs are now on top and the timeline is on the bottom, there’s no reason, I just think that that layout is more aesthetic.

And I might name it something other than the ajax animator, something just like Animator or something


I GOT A GOOGLE WAVE INVITE! 13 August 2009

IM STILL WAITING TO GET MY ACCOUNT BUT I JUST GOT AN INVITE AND IM SOOO HAPPY!!!!!!!!!!! SEE? LOOK I’M USING CAPS LOCK TO EXPRESS MY HAPPYNESS! IT’S REALLY REALLY REALLY AWESOME EVEN THOUGH I HAVENT ACTUALLY TRIED IT OUT YET! BUT ITS SOOOO COOL! AND LETS BE HONEST, AT LEAST THIS POST ISNT IN COMIC SANS MS.

EDIT: i got my wave account. It’s cool, but there’s nobody to talk to.


VectorEditor on Wave 12 August 2009

So I don’t actually have wave yet, but for all 2 of you (likely less) who have used pygowave, an open source third party implementation of the wave protocol. So there was how I developed it, I read through the APIs and tested on pygowave. So what does VectorEditor do that svg-edit and… erm… svg-edit don’t do?

First, VectorEditor for wave is really really real time. Waaay more real time than svg-edit (not really). But VectorEdit (VectorWave might be a nice name.. I’m going to try using that name from now on in this post) transmits the data such as even while the shape is still being drawn, rotated, or moved (rotation needs work). Another nice feature is that the transitions are animated so things are even more seamless.

Another important feature is shape locking. So when someone selects a shape, it gets locked and can’t be edited by anyone else. If anyone tries, an alert box appears saying “Shape shape:5sdfwef98dfe3ssdf is locked by user antimatter15@pygowave.p2k-network.org”. svg-edit (the latter) doesn’t support moving things after they’re created so it doesn’t really matter then, and I’m quite certain the former doesn’t do any type of locking.

And lets not forget the likely most important, yet totally untested feature that seperates VectorEditor from the rest: IE support, which is inherent since it uses Raphael for rendering, but it may not be necessary since google may be making some shim-type system of hacking svg awesomeness onto IE and making the whole VectorEditor project useless.

So if you want to try it out, go to pygowave, sign up, create a wave and add the